Hybrid active learning for cross%2Ddomain video concept detection.

ACM international workshop on very%2Dlarge%2Dscale multimedia corpus

Explicit and implicit concept%2Dbased video retrieval with bipartite graph propagation model.

Explicit and implicit concept%2Dbased video retrieval with bipartite graph propagation model.

Comparing Evaluation Protocols on the KTH Dataset.

Controlling your TV with gestures.
Vision-based user interfaces enable natural interaction modalities such as gestures. Such interfaces require computationally intensive video processing at low latency. We demonstrate an application that recognizes gestures to control TV operations. Accurate recognition is achieved by using a new descriptor called MoSIFT, which explicitly encodes optical flow with appearance features. MoSIFT is computationally expensive — a sequential implementation runs 100 times slower than real time. To reduce latency sufficiently for interaction, the application is implemented on a runtime system that exploits the parallelism inherent in video understanding applications. Categories and Subject Descriptors C.3 [Computer Systems Organization]: Special-Purpose and application base systems. D.2 [Software] Software engineering.
A Multi%2DPronged Approach to Improving Semantic Extraction of News Video.

ACM SIGMM the first workshop on web%2Dscale multimedia corpus %28WSMC09%29.

ACM SIGMM the first workshop on web%2Dscale multimedia corpus %28WSMC09%29.

Identifying news videos%27 ideological perspectives using emphatic patterns of visual concepts.

Video Content Analysis.
The principal goal of this proposal is to design an automatic video processing architecture that will enable us to design and develop efficient multimedia applications such as summarization, indexing, retrieval and others. While there are many research projects in this area, many, if not all, rely on MPEG-7 for high-level semantic retrieval. The proposed architecture is compatible with MPEG-7 in that the proposed architecture can be annotated using the MPEG-7 standard, but we emphasize on user relevance feedback in the high-level semantic retrieval task. The proposed research is built upon a substantial body of preliminary work that serves as proof of concept for the proposed architecture and other techniques. The preliminary work includes the development of automatic clustering technique using Delaunay Triangulation, automatic video summary generation and automatic shot boundary detection. We address the problem of automatic video indexing and organization and video information retrieval through user interaction at multiple levels. Our work focuses on the following three important areas: (1) temporal video segmentation; (2) construction of
Semantic Multimedia
Abstract. Multimedia constitutes an interesting field of application for Semantic Web and Semantic Web reasoning, as the access and management of multimedia content and context depends strongly on the semantic descriptions of both. At the same time, multimedia resources constitute complex objects, the descriptions of which are involved and require the foundation on sound modeling practice in order to represent findings of low- and high level multimedia analysis and to make them accessible via Semantic Web querying of resources. This tutorial aims to provide a red thread through these different issues and to give an outline of where Semantic Web modeling and reasoning needs to further contribute to the area of semantic multimedia for the fruitful interaction between these two fields of computer science. 1 Semantics for Multimedia Multimedia objects are ubiquitous, whether found via web search (e.g., Google 3 or Yahoo! 4 images), or via dedicated sites (e.g., Flickr 5 or YouTube 6)orinthe
Efficient search the informedia video retrieval system.

%28Un%29Reliability of video concept detection.
Great effort has been made to improve video concept detection and continuous progress has been reported. With the current evaluation method being confined to carefully annotated domains and thus quite forgiving, the reliability of the state-of-the-art concept classifiers remains in question. Adopting a more rigorous evaluation approach, we find that most concept classifiers built using the mainstream approach are unreliable because they generalize poorly to domains other than their training domain. Moreover, evidences show that SVM-based concept classifiers learn little beyond memorizing most of the positive training data, and behave close to memory-based models such as kNN indicated by comparable performance between the two models. Examining the properties of the reliable concept classifiers, we find that the classifiers of frequent concepts, “bloated ” classifiers, and classifiers capable of learning the pattern of data, tend to be more reliable. This paper contributes to a better understanding of concept detection, suggests heuristics to identify reliable concept classifiers, and discusses solutions to improving concept detection reliability.

A Joint Topic and Perspective Model for Ideological Discourse.
Abstract. Polarizing discussions on political and social issues are common in mass and user-generated media. However, computer-based understanding of ideological discourse has been considered too difficult to undertake. In this paper we propose a statistical model for ideology discourse. By ideology we mean “a set of general beliefs socially shared by a group of people. ” For example, Democratic and Republican are two major political ideologies in the United States. The proposed model captures lexical variations due to an ideological text’s topic and due to an author or speaker’s ideological perspective. To cope with the non-conjugacy of the logistic-normal prior we derive a variational inference algorithm for the model. We evaluate the proposed model on synthetic data as well as a written and a spoken political discourse. Experimental results strongly support that ideological perspectives are reflected in lexical variations. 1

Do These News Videos Portray a News Event from Different Ideological Perspectives%3F.

Vox Populi Annotation Measuring Intensity of Ideological Perspectives by Aggregating Group Judgments.

A framework for classifier adaptation and its applications in concept detection.
There is often a need to adapt supervised classifiers such as semantic concept detectors across different domains of data. This paper describes a generic framework for function-level classifier adaptation based on regularized loss minimization. It directly modifies the decision function of an existing classifier of any type into a classifier for a new domain, based on limited labeled data in the new domain and no “old data”, which makes it an efficient and flexible framework. We then extend this framework to adapt multiple classifiers into one classifier, with the weights of existing classifiers learned automatically to reflect their utility. We elaborate on two concrete adaptation algorithms derived from the framework, namely adaptive SVM and multi-adaptive SVM, for one-toone and many-to-one adaptation respectively. In the experiments of adapting semantic concept detectors across video channels/types, our adaptation approach is proven to be superior to using original (unadapted) classifiers or building new ones in terms of accuracy and labeling effort.
Informedia %40 TRECVID2008 Exploring New Frontiers.

Exploring the utility of fast%2Dforward surrogates for bbc rushes.
This paper discusses in detail our approaches for producing the video summaries submitted to the TRECVID 2008 BBC rushes summarization task, including the baseline method. Empirical work produced during and after the TRECVID 2007 rushes summarization task gave strong evidence that a simple 50x method (sampling every 50 th frame) provides excellent coverage (text inclusion performance). Our submissions for TRECVID 2008 investigated the effects of junk frame removal, including a comprehensible audio track, and emphasizing pans and zooms when backfilling to reclaim the space removed with the noise shots from the original 50x set. Results show that 50x based methods provide excellent coverage as expected. There were limited effects for the other strategies to improve user satisfaction, with the discussion providing some insights for future video summary development and evaluation work.
Measuring novelty and redundancy with multiple modalities in cross%2Dlingual broadcast news.

VideOlympics Real%2DTime Evaluation of Multimedia Retrieval Systems.
Interactive prototypes are often the best way to convince an audience of a new multimedia technology’s possible impact. Because of its dynamic audiovisual nature, a multimedia application demonstration communicates applied science more effectively than a static description in a journal publication would. Ideally, a multimedia demonstrator grasps the audience’s attention by presenting effective results, advanced multimodal interfaces, novel means of user interaction, or combinations thereof. An interactive demo offers researchers a means to engage their audience in a way that they could never achieve with a written manuscript alone. All major multimedia conferences have adopted demo sessions where researchers, equipped with their laptop-installed system, show their demo to the conference attendees on a one-on-one basis. While effective, these demo sessions lack a common denominator. Individual systems provide solutions for different tasks on different data sets, and the conference attendees have to become familiar with the peculiarities of each individual demo. Moreover, a one-on-one demo session makes it infeasible to present a system to the entire conference audience. In addition, due to their lack of focus, the impact of demo sessions on the audience is suboptimal. Editor’s Note Video search is an experience for the senses. As a result, traditional information retrieval metrics can’t fully measure the quality of a video search system. To provide a more interactive assessment of today’s video search engines, the authors have organized the VideOlympics as a real-time evaluation showcase where systems compete to answer specific video searches in front of a live audience. At VideOlympics, seeing and hearing is believing. —John R. Smith Establishing focus A good way to establish focus is letting several demo systems solve the same problem on a similar data set. A notable example of such a focused effort takes place at the demo session at the National Institute for Standards
Multimodal News Story Clustering With Pairwise Visual Near%2DDuplicate Constraint.

Novelty detection for cross%2Dlingual news stories with visual duplicates and speech transcripts.
An overwhelming volume of news videos from different channels and languages is available today, which demands automatic management of this abundant information. To effectively search, retrieve, browse and track cross-lingual news stories, a news story similarity measure plays a critical role in assessing the novelty and redundancy among them. In this paper, we explore the novelty and redundancy detection with visual duplicates and speech transcripts for cross-lingual news stories. News stories are represented by a sequence of keyframes in the visual track and a set of words extracted from speech transcript in the audio track. A major difference to pure text documents is that the number of keyframes in one story is relatively small compared to the number of words and there exist a large number of non-near-duplicate keyframes. These features make the behavior of similarity measures different compared to traditional textual collections. Furthermore, the textual features and visual features complement each other for news stories. They can be further combined to boost the performance. Experiments on the TRECVID-2005 cross-lingual news video corpus show that approaches on textual features and visual features demonstrate different performance, and measures on visual features are quite effective. Overall, the cosine distance on keyframes is still a robust measure. Language models built on visual features demonstrate promising performance. The fusion of textual and visual features improves overall performance.
Cross%2Ddomain video concept detection using adaptive svms.

Practical elimination of near%2Dduplicates from web video search.

Query expansion using probabilistic local feedback with application to multimedia retrieval.

How many high%2Dlevel concepts will fill the semantic gap in news video retrieval%3F
video retrieval?
Adapting SVM Classifiers to Data with Shifted Distributions.
Many data mining applications can benefit from adapting existing classifiers to new data with shifted distributions. In this paper, we present Adaptive Support Vector Machine (Adapt-SVM) as an efficient model for adapting a SVM classifier trained from one dataset to a new dataset where only limited labeled examples are available. By introducing a new regularizer into SVM’s objective function, Adapt-SVM aims to minimize both the classification error over the training examples, and the discrepancy between the adapted and original classifier. We also propose a selective sampling strategy based on the loss minimization principle to seed the most informative examples for classifier adaptation. Experiments on an artificial classification task and on a benchmark video classification task shows that Adapt-SVM outperforms several baseline methods in terms of accuracy and/or efficiency. 1
Adapting SVM Classifiers to Data with Shifted Distributions.
Many data mining applications can benefit from adapting existing classifiers to new data with shifted distributions. In this paper, we present Adaptive Support Vector Machine (Adapt-SVM) as an efficient model for adapting a SVM classifier trained from one dataset to a new dataset where only limited labeled examples are available. By introducing a new regularizer into SVM’s objective function, Adapt-SVM aims to minimize both the classification error over the training examples, and the discrepancy between the adapted and original classifier. We also propose a selective sampling strategy based on the loss minimization principle to seed the most informative examples for classifier adaptation. Experiments on an artificial classification task and on a benchmark video classification task shows that Adapt-SVM outperforms several baseline methods in terms of accuracy and/or efficiency. 1


Exploring Concept Selection Strategies for Interactive Video Search.
Ranked shot lists from 39 automated LSCOM-Lite concept classifiers are investigated with respect to 24 TRECVID 2006 topics. Selecting the best fitting concept or pair of concepts produces the shot set with greatest utility, rather than drawing fewer shots from a larger set of concepts. Mean average precision measures show concept-based shot sets have great utility for topics when perfectly traversed by a user. Using empirical data, however, shows that realistic ability to separate relevant shots from irrelevant ones and recall all the relevant ones is topic-dependent and far from perfect. Concept-based strategies including user-driven selection strategies not using idealized oracle prioritization are also discussed, with implications for query-by-concept in interactive video retrieval as concept spaces grow from tens to thousands. 1.
A Hybrid Approach to Improving Semantic Extraction of News Video.
In this paper we describe a hybrid approach to improving semantic extraction from news video. Experiments show the value of careful parameter tuning, exploiting multiple feature sets and multilingual linguistic resources, applying text retrieval approaches for image features, and establishing synergy between multiple concepts through undirected graphical models. No single approach provides a consistently better result for every concept detection, which suggests that extracting video semantics should exploit multiple resources and techniques rather than a single approach. 1.
Evaluating bag%2Dof%2Dvisual%2Dwords representations in scene classification.

Discriminative Fields for Modeling Semantic Concepts in Video.
According to some current thinking, a very large number of semantic concepts could provide researcher a novel way to characterize video and be utilized for video retrieval and understanding. These semantic concepts do not isolate to each other and thus exploiting relationships between multiple semantic concepts in video could be a very useful source to enhance the concept detection performance. In this paper we present a discriminative learning framework called Multi-concept Discriminative Random Field (MDRF) for building probabilistic models on video semantic concept detections by incorporating related concepts as well as the observation. The proposed model exploits the power of discriminative graphical models to simultaneously capture the associations of concept with observed data and the interactions between related concepts. Compared with previous methods, this model can not only capture the co-occurrence between concepts but also incorporate the data observation in a unified framework. We also present an approximate parameter estimation algorithm and apply it to TRECVID 2005 data. Our experiments show promising results compared to the single concept learning approach for video semantic detection. 1.
Harmonium Models for Semantic Video Representation and Classification.
Accurate and efficient video classification demands the fusion of multimodal information and the use of intermediate representations. Combining the two ideas into the same framework, we propose a probabilistic approach for video classification using intermediate semantic representations derived from the multi-modal features. Based on a class of bipartite undirected graphical models named harmonium, our approach represents video data as latent semantic topics derived by jointly modeling the transcript keywords and color-histogram features, and perform classification using these latent topics under a unified framework. We show satisfactory classification performance of our approach on a benchmark dataset, and some interesting insights of the data provided by this approach.
Summarizing BBC Rushes the Informedia Way.

Clever clustering vs. simple speed%2Dup for summarizing rushes.

Can High%2DLevel Concepts Fill the Semantic Gap in Video Retrieval%3F A Case Study With Broadcast News.

A review of text and image retrieval approaches for broadcast news video.

Are These Documents Written from Different Perspectives%3F A Test of Different Perspectives Based on Statistical Distribution Divergence.
In this paper we investigate how to automatically determine if two document collections are written from different perspectives. By perspectives we mean a point of view, for example, from the perspective of Democrats or Republicans. We propose a test of different perspectives based on distribution divergence between the statistical models of two collections. Experimental results show that the test can successfully distinguish document collections of different perspectives from other types of collections. 1
Extreme video retrieval joint maximization of human and computer performance.
We present an efficient system for video search that maximizes the use of human bandwidth, while at the same time exploiting the machine’s ability to learn in real-time from user selected relevant video clips. The system exploits the human capability for rapidly scanning imagery augmenting it with an active learning loop, which attempts to always present the most relevant material based on the current information. Two versions of the human interface were evaluated, one with variable page sizes and manual paging, the other with a fixed page size and automatic paging. Both require absolute attention and focus of the user for optimal performance. In either case, as humans search and find relevant results, the system can invisibly re-rank its previous best guesses using a number of knowledge sources, such as image similarity, text similarity, and temporal proximity. Experimental evidence shows a significant improvement using the combined extremes of human and machine power over either approach alone.
3WNews who%2C where%2C and when in news video.

Efficient Margin%2DBased Rank Learning Algorithms for Information Retrieval.
Abstract. Learning a good ranking function plays a key role for many applications including the task of (multimedia) information retrieval. While there are a few rank learning methods available, most of them need to explicitly model the relations between every pair of relevant and irrelevant documents, and thus result in an expensive training process for large collections. The goal of this paper is to propose a general rank learning framework based on the margin-based risk minimization principle and develop a set of efficient rank learning approaches that can model the ranking relations with much less training time. Its flexibility allows a number of margin-based classifiers to be extended to their rank learning counterparts such as the ranking logistic regression developed in this paper. Experimental results show that this efficient learning algorithm can successfully learn a highly effective retrieval function for multimedia retrieval on the TRECVID’03-’05 collections. 1 1
Efficient Margin%2DBased Rank Learning Algorithms for Information Retrieval.
Abstract. Learning a good ranking function plays a key role for many applications including the task of (multimedia) information retrieval. While there are a few rank learning methods available, most of them need to explicitly model the relations between every pair of relevant and irrelevant documents, and thus result in an expensive training process for large collections. The goal of this paper is to propose a general rank learning framework based on the margin-based risk minimization principle and develop a set of efficient rank learning approaches that can model the ranking relations with much less training time. Its flexibility allows a number of margin-based classifiers to be extended to their rank learning counterparts such as the ranking logistic regression developed in this paper. Experimental results show that this efficient learning algorithm can successfully learn a highly effective retrieval function for multimedia retrieval on the TRECVID’03-’05 collections. 1 1
Annotating News Video with Locations.
Abstract. The location of video scenes is an important semantic descriptor especially for broadcast news video. In this paper, we propose a learning-based approach to annotate shots of news video with locations extracted from video transcript, based on features from multiple video modalities including syntactic structure of transcript sentences, speaker identity, temporal video structure, and so on. Machine learning algorithms are adopted to combine multi-modal features to solve two sub-problems: (1) whether the location of a video shot is mentioned in the transcript, and if so, (2) among many locations in the transcript, which are correct one(s) for this shot. Experiments on TRECVID dataset demonstrate that our approach achieves approximately 85 % accuracy in correctly labeling the location of any shot in news video. 1
Exploring the Synergy of Humans and Machines in Extreme Video Retrieval.

Label Disambiguation and Sequence Modeling for Identifying Human Activities from Wearable Physiological Sensors.

Mining Relationship Between Video Concepts using Probabilistic Graphical Models.
For large scale automatic semantic video characterization, it is necessary to learn and model a large number of semantic concepts. These semantic concepts do not exist in isolation to each other and exploiting this relationship between multiple video concepts could be a useful source to improve the concept detection accuracy. In this paper, we describe various multi-concept relational learning approaches via a unified probabilistic graphical model representation and propose using numerous graphical models to mine the relationship between video concepts that have not been applied before. Their performances in video semantic concept detection are evaluated and compared on two TRECVID’05 video collections. 1.
Which Thousand Words are Worth a Picture%3F Experiments on Video Retrieval using a Thousand Concepts.

Exploring temporal consistency for video analysis and retrieval.
Temporal consistency is ubiquitous in video data, where temporally adjacent video shots usually share similar visual and semantic content. This paper presents a thorough study of temporal consistency defined with respect to semantic concepts and query topics using quantitative measures, and discusses its implications to video analysis and retrieval tasks. We further show that, in interactive settings, using temporal consistency leads to considerable improvement on the performance of semantic concept detection and retrieval of video data. Specifically, an active learning method with temporal sampling strategy is proposed for building classifiers of semantic concepts, and a temporal reranking method is proposed for improving the efficiency of interactive video search. Both methods outperform existing methods by considerable margins on the TRECVID dataset.
Diversity in multimedia information retrieval research.
Multimedia information retrieval is a highly diverse field. A variety of data types, research problems, methodologies are involved. Researchers in the field come from very different disciplines, ranging from mathematical and physical sciences, computational sciences and engineering, to application domains. The panel, consisting of highly visible active researchers from both academia and the industry, opens a discussion on the importance of diversity to the healthy growth of the field. This paper records their opinions expressed at the panel.
A Discriminative Learning Framework with Pairwise Constraints for Video Object Classification.
In video object classification, insufficient labeled data may at times be easily augmented with pairwise constraints on sample points, i.e, whether they are in the same class or not. In this paper, we proposed a discriminative learning approach which incorporates pairwise constraints into a conventional margin-based learning framework. The proposed approach offers several advantages over existing approaches dealing with pairwise constraints. First, as opposed to learning distance metrics, the new approach derives its classification power by directly modeling the decision boundary. Second, most previous work handles labeled data by converting them to pairwise constraints and thus leads to much more computation. The proposed approach can handle pairwise constraints together with labeled data so that the computation is greatly reduced. The proposed approach is evaluated on a people classification task with two surveillance video datasets.
Probabilistic latent query analysis for combining multiple retrieval sources.
Combining the output from multiple retrieval sources over the same document collection is of great importance to a number of retrieval tasks such as multimedia retrieval, web retrieval and meta-search. To merge retrieval sources adaptively according to query topics, we propose a series of new approaches called probabilistic latent query analysis (pLQA), which can associate non-identical combination weights with latent classes underlying the query space. Compared with previous query independent and query-class based combination methods, the proposed approaches have the advantage of being able to discover latent query classes automatically without using prior human knowledge, to assign one query to a mixture of query classes, and to determine the number of query classes under a model selection principle. Experimental results on two retrieval tasks, i.e., multimedia retrieval and meta-search, demonstrate that the proposed methods can uncover sensible latent classes from training data, and can achieve considerable performance gains.
Large%2DScale Concept Ontology for Multimedia.
The DTO-sponsored LSCOM workshop has developed a large concept lexicon for multimedia which includes approximately 1000 concepts related events, objects, locations, people, and programs [1, 3]. Human subjects annotated 449 of those concepts over a corpus of 80 hours of TRECVID 2005 videos [2], composed of 61,901 subshots, by visually inspecting a keyframe, which is a single still image, from each subshot and making a judgment as to whether a given concept is present or absent within that keyframe. This keyframe-only approach is fast and has proven to be sufficient for many of the static concepts, such as “airplane, ” “car, ” or “person; ” however, event or activity concepts, like “airplane taking off, ” “car driving,” or “person walking, ” have a temporal component and are difficult for humans to judge, given only a single keyframe. It is necessary, therefore, to acquire labels based on a viewing of the video in motion to truly make a reliable annotation for the event. From the total 449 concepts that were annotated using the keyframe-based method, we selected 24 to be re-annotated using a video-based approach. These videobased annotations for 24 concepts are freely available for download, along with the keyframe-based labels for all 449 concepts [4]. To gather these video-based judgments, we designed a simple tool for providing binary positive/negative labels based on viewing a video clip of a subshot. The tool is web-based and can be used remotely with
A Discriminative Learning Framework with Pairwise Constraints for Video Object Classification.
In video object classification, insufficient labeled data may at times be easily augmented with pairwise constraints on sample points, i.e, whether they are in the same class or not. In this paper, we proposed a discriminative learning approach which incorporates pairwise constraints into a conventional margin-based learning framework. The proposed approach offers several advantages over existing approaches dealing with pairwise constraints. First, as opposed to learning distance metrics, the new approach derives its classification power by directly modeling the decision boundary. Second, most previous work handles labeled data by converting them to pairwise constraints and thus leads to much more computation. The proposed approach can handle pairwise constraints together with labeled data so that the computation is greatly reduced. The proposed approach is evaluated on a people classification task with two surveillance video datasets.
Learning rich semantics from news video archives by style analysis.
We propose a generic and robust framework for news video indexing which we founded on a broadcast news production model. We identify within this model four production phases, each providing useful metadata for annotation. In contrast to semiautomatic indexing approaches which exploit this information at production time, we adhere to an automatic data-driven approach. To that end, we analyze a digital news video using a separate set of multimodal detectors for each production phase. By combining the resulting production-derived features into a statistical classifier ensemble, the framework facilitates robust classification of several rich semantic concepts in news video; rich meaning that concepts share many similarities in their production process. Experiments on an archive of 120 hours of news video from the 2003 TRECVID benchmark show that a combined analysis of production phases yields the best results. In addition, we demonstrate that the accuracy of the proposed style analysis framework for classification of several rich semantic concepts is state-of-the-art.
Multiple instance learning for labeling faces in broadcasting news video.

Putting active learning into multimedia applications dynamic definition and refinement of concept classifiers.
The authors developed an extensible system for video exploitation that puts the user in control to better accommodate novel situations and source material. Visually dense displays of thumbnail imagery in storyboard views are used for shot-based video exploration and retrieval. The user can identify a need for a class of audiovisual detection, adeptly and fluently supply training material for that class, and iteratively evaluate and improve the resulting automatic classification produced via multiple modality active learning and SVM. By iteratively reviewing the output of the classifier and updating the positive and negative training samples with less effort than typical for relevance feedback systems, the user can play an active role in directing the classification process while still needing to truth only a very small percentage of the multimedia data set. Examples are given illustrating the iterative creation of a classifier for a concept of interest to be included in subsequent investigations, and for a concept typically deemed irrelevant to be weeded out in follow-up queries. Filtering and browsing tools making use of existing and iteratively added concepts put the user further in control of the multimedia browsing and retrieval process.
Lessons for the Future from a Decade of Informedia Video Analysis Research.

The Use and Utility of High%2DLevel Semantic Features in Video Retrieval.
Abstract. This paper investigates the applicability of high-level semantic features for video retrieval using the benchmarked data from TRECVID 2003 and 2004, addressing the contributions of features like outdoor, face, and animal in retrieval, and if users can correctly decide on which features to apply for a given need. Pooled truth data gives evidence that some topics would benefit from features. A study with 12 subjects found that people often disagree on the relevance of a feature to a particular topic, including disagreement within the 8 % of positive feature-topic associations strongly supported by truth data. When subjects concur, their judgments are correct, and for those 51 topicfeature pairings identified as significant we conduct an investigation into the best interactive search submissions showing that for 29 pairs, topic performance would have improved had users had access to ideal classifiers for those features. The benefits derive from generic features applied to generic topics (27 pairs), and in one case a specific feature applied to a specific topic. Re-ranking submitted shots based on features shows promise for automatic search runs, but not for interactive runs where a person already took care to rank shots well. 1
Assessing Effectiveness in Video Retrieval.
Abstract. This paper examines results from the last two years of the TRECVID video retrieval evaluations. While there is encouraging evidence about progress in video retrieval, there are several major disappointments confirming that the field of video retrieval is still in its infancy. Many publications blithely attribute improvements in retrieval tasks to the different techniques without paying much attention to the statistical reliability of the comparisons. We conduct an analysis of the official TRECVID evaluation results, using both retrieval experiment error rates and ANOVA measures, and demonstrate that the difference between many systems is not statistically significant. We conclude the paper with the lessons learned from both results with and without statistically significant difference. 1
Multi%2Dmodal analysis for person type classification in news video.
Classifying the identities of people appearing in broadcast news video into anchor, reporter, or news subject is an important topic in high-level video analysis. Given the visual resemblance of different types of people, this work explores multi-modal features derived from a variety of evidences, such as the speech identity, transcript clues, temporal video structure, named entities, and uses a statistical learning approach to combine all the features for person type classification. Experiments conducted on ABC World News Tonight video have demonstrated the effectiveness of the approach, and the contributions of different categories of features have been compared. Keywords: Multi-modality analysis, person type classification, broadcast news video 1.
Revisiting the effect of topic set size on retrieval error.

Multi%2Dmodal analysis for person type classification in news video.
Classifying the identities of people appearing in broadcast news video into anchor, reporter, or news subject is an important topic in high-level video analysis. Given the visual resemblance of different types of people, this work explores multi-modal features derived from a variety of evidences, such as the speech identity, transcript clues, temporal video structure, named entities, and uses a statistical learning approach to combine all the features for person type classification. Experiments conducted on ABC World News Tonight video have demonstrated the effectiveness of the approach, and the contributions of different categories of features have been compared. Keywords: Multi-modality analysis, person type classification, broadcast news video 1.
Mining Associated Text and Images with Dual%2DWing Harmoniums.
We propose a multi-wing harmonium model for mining multimedia data that extends and improves on earlier models based on two-layer random fields, which capture bidirectional  dependencies between hidden  topic aspects and observed inputs. This model can be viewed as an undirected counterpart of the two-layer directed models such  as LDA for similar tasks, but bears significant  di#erence in inference/learning cost tradeoffs, latent topic representations, and topic  mixing mechanisms. In particular, our model facilitates efficient inference and robust topic  mixing, and potentially provides high flexibilities  in modeling the latent topic spaces. A  contrastive divergence and a variational algorithm  are derived for learning. We specialized  our model to a dual-wing harmonium for  captioned images, incorporating a multivariate  Poisson for word-counts and a multivariate  Gaussian for color histogram. We present  empirical results on the applications of this  model to classification, retrieval and image  annotation on news video collections, and we  report an extensive comparison with various  extant models.
Learning query%2Dclass dependent weights in automatic video retrieval.
Combining retrieval results from multiple modalities plays a crucial role for video retrieval systems, especially for automatic video retrieval systems without any user feedback and query expansion. However, most of current systems only utilize query independent combination or rely on explicit user weighting. In this work, we propose using query-class dependent weights within a hierarchial mixture-of-expert framework to combine multiple retrieval results. We first classify each user query into one of the four predefined categories and then aggregate the retrieval results with query-class associated weights, which can be learned from the development data efficiently and generalized to the unseen queries easily. Our experimental results demonstrate that the performance with query-class dependent weights can considerably surpass that with the query independent weights.
Multimedia information retrieval workshop report.

Proceedings MDDE %2704

Learning query%2Dclass dependent weights in automatic video retrieval.
Combining retrieval results from multiple modalities plays a crucial role for video retrieval systems, especially for automatic video retrieval systems without any user feedback and query expansion. However, most of current systems only utilize query independent combination or rely on explicit user weighting. In this work, we propose using query-class dependent weights within a hierarchial mixture-of-expert framework to combine multiple retrieval results. We first classify each user query into one of the four predefined categories and then aggregate the retrieval results with query-class associated weights, which can be learned from the development data efficiently and generalized to the unseen queries easily. Our experimental results demonstrate that the performance with query-class dependent weights can considerably surpass that with the query independent weights.
Naming every individual in news video monologues.
Naming every individual person appearing in broadcast news videos with names detected from the video transcript leads to better access of the news video content. In this paper, we approach this challenging problem with a statistical learning method. Two categories of information extracted from multiple video modalities have been explored, namely features, which help distinguish the true name of every person, as well as constraints, which reveal the relationships among the names of different persons. The personnaming problem is formulated into a learning framework which predicts the most likely name for each person based on the features, and refines the predictions using the constraints. Experiments conducted on ABC World New Tonight and CNN Headline News videos demonstrate that this approach outperforms a nonlearning alternative by a large amount.
Successful approaches in the TREC video retrieval evaluations.
This paper reviews successful approaches in evaluations of video retrieval over the last three years. The task involves the search and retrieval of shots from MPEG digitized video recordings using a combination of automatic speech, image and video analysis and information retrieval technologies. The search evaluations are grouped into interactive (with a human in the loop) and noninteractive (where the human merely enters the query into the system) submissions. Most non-interactive search approaches have relied extensively on text retrieval, and only recently have image-based features contributed reliably to improved search performance. Interactive approaches have substantially outperformed all non-interactive approaches, with most systems relying heavily on the user’s ability to refine queries and reject spurious answers. We will examine both the successful automatic search approaches and the user interface techniques that have enabled high performance video retrieval.
What%27s News%2C What%27s Not%3F Associating News Videos with Words.
Text retrieval from broadcast news video is unsatisfactory,  because a transcript word frequently does not directly `describe&#039; the shot  when it was spoken. Extending the retrieved region to a window around  the matching keyword provides better recall, but low precision. We improve  on text retrieval using the following approach: First we segment  the visual stream into coherent story-like units, using a set of visual news  story delimiters. After filtering out clearly irrelevant classes of shots, we  are still left with an ambiguity of how words in the transcript relate  to the visual content in the remaining shots of the story. Using a limited  set of visual features at di#erent semantic levels ranging from color  histograms, to faces, cars, and outdoors, an association matrix captures  the correlation of these visual features to specific transcript words. This  matrix is then refined using an EM approach. Preliminary results show  that this approach has the potential to significantly improve retrieval  performance from text queries.
Finding Person X Correlating Names with Visual Appearances.
Abstract. People as news subjects carry rich semantics in broadcast news video and therefore finding a named person in the video is a major challenge for video retrieval. This task can be achieved by exploiting the multi-modal information in videos, including transcript, video structure, and visual features. We propose a comprehensive approach for finding specific persons in broadcast news videos by exploring various clues such as names occurred in the transcript, face information, anchor scenes, and most importantly, the timing pattern between names and people. Experiments on the TRECVID 2003 dataset show that our approach achieves high performance. 1
Co%2Dretrieval A Boosted Reranking Approach for Video Retrieval.


Towards a Large Scale Concept Ontology for Broadcast Video.

A Discriminative Learning Framework with Pairwise Constraints for Video Object Classification.
In video object classification, insufficient labeled data may at times be easily augmented with pairwise constraints on sample points, i.e, whether they are in the same class or not. In this paper, we proposed a discriminative learning approach which incorporates pairwise constraints into a conventional margin-based learning framework. The proposed approach offers several advantages over existing approaches dealing with pairwise constraints. First, as opposed to learning distance metrics, the new approach derives its classification power by directly modeling the decision boundary. Second, most previous work handles labeled data by converting them to pairwise constraints and thus leads to much more computation. The proposed approach can handle pairwise constraints together with labeled data so that the computation is greatly reduced. The proposed approach is evaluated on a people classification task with two surveillance video datasets.

Detection of TV news monologues by style analysis.
We propose a method for detection of semantic concepts in produced video based on style analysis. Recognition of concepts is done by applying a classifier ensemble to the detected style elements. As a case study we present a method for detecting the concept of news subject monologues. Our approach had the best average precision performance amongst 26 submissions in the 2003 TRECVID benchmark. 1.
Towards robust face recognition from multiple views.
This paper presents a novel approach to aid face recognition: Using multiple views of a face, we construct a 3D model instead of directly using the 2D images for recognition. Our framework is designed for videos, which contain many instances of a target face from a sequence of slightly differing views, as opposed to a single static picture of the face. Specifically, we reconstruct the 3D face shapes from two orthogonal views and select features based on pairwise distances between landmark points on the model using Fisher&#039;s Linear Discriminant. While 3D face shape reconstruction is sensitive to the quality of the feature point localization, our experiments show that 3D reconstruction together with the regularized Fisher&#039;s Linear Discriminant can provide highly accurate face recognition from multiple facial views. Experiments on the Carnegie Mellon PIE (Pose, Illumination and Expressions) database containing 68 people’s faces with at least 3 expressions under varying lighting conditions demonstrate vastly improved performance 1.
Merging rank lists from multiple sources in video classification.
Multimedia corpora increasingly consist of data from multiple sources, with di#erent characteristics that can be exploited by specialized applications. This paper focuses on video classification over multiple-source collections, and addresses the question whether classifiers should train from individual sources or from a full data set across all sources. If training separately, how can rank lists from di#erent sources be merged e#ectively? We formulate the problem of merging ranked lists as learning a function mapping from local scores to global scores, and propose a learning method based on logistic regression. In our experiments we find that source characteristics are very important for video classification. Moreover, our method of learning mapping functions perform significant better than merging methods without explicitly learning the mapping functions.
Modeling timing features in broadcast news video classification.
Broadcast news programs are well-structured video, and timing can be a strong predictor for specific types of news reports. However, learning a classifier using timing features may not be an easy task when training data are noisy. In this paper, we approach the problem from the generative model perspective, and approximate the class density in a non-parametric fashion. The results show that timing is a simple but extremely e#ective feature, and our method can achieve significantly better performance than a discriminative classifier.
Multi%2Dclass active learning for video semantic feature extraction.
Active learning has been demonstrated to be a useful tool to reduce human labeling effort for many multimedia applications especially for those handling large video collections. However, most of the previous work on active learning only focuses on binary classification, which greatly limits the applicability of active learning. In this paper, we present a multi-class active learning approach which extends active learning from binary classification to multi-class classification using a unified representation with margin-based loss functions. The experimental results on the TREC03 semantic feature extraction task shows that the proposed active learning approach works effectively even with a significantly reduced amount of labeled data. 1.
Dining Activity Analysis Using a Hidden Markov Model.
We describe an algorithm for dining activity analysis in a nursing home. Based on several features, including motion vectors and distance between moving regions in the subspace of an individual person, a hidden Markov model is proposed to characterize different stages in dining activities with certain temporal order. Using HMM model, we are able to identify the start (and ending) of individual dining events with high accuracy and low false positive rate. This approach could be successful in assisting caregivers in assessments of resident&#039;s activity levels over time. 1.
Multi%2Dmodal classification in digital news libraries.

Video grammar for locating named people.
Finding a named person in broadcast news video is important in video retrieval. Relying on the text information such as video transcript and OCR text, this task suffers from the temporal mismatch between a person&#039;s visual appearance and the occurrence of his/her name in the text. By exploring video grammar regarding the concurrence pattern between faces and names, we propose an extended text-based IR method to overcome this problem, which yield superior performance. 1.
The combination limit in multimedia retrieval.
Combining search results from multimedia sources is crucial for dealing with heterogeneous multimedia data, particularly in multimedia retrieval where a final ranked list of items of interest is returned sorted by confidence or relevance. However, relatively little attention has been given to combination functions, especially their upper bound performance limits. This paper presents a theoretical framework for studying upper bounds for two types of combination functions. A general upper bound and two approximations are proposed for monotonic combination functions. We also studied the upper bounds for linear combination functions using a global optimization technique. Our experimental results show that the choice of combination functions has a considerable influence to retrieval performance. 1.
Negative pseudo%2Drelevance feedback in content%2Dbased video retrieval.
Video information retrieval requires a system to find information relevant to a query which may be represented simultaneously in di#erent ways through a text description, audio, still images and/or video sequences. We present a novel approach that uses pseudo-relevance feedback from retrieved items that are NOT similar to the query items without further inquiring user feedback. We provide insight into this approach using a statistical model and suggest a score combination scheme via posterior probability estimation. An evaluation on the 2002 TREC Video Track queries shows that this technique can improve video retrieval performance on a real collection. We believe that negative pseudo-relevance feedback shows great promise for very difficult multimedia retrieval tasks, especially when combined with other di#erent retrieval algorithms.
Multimedia Search with Pseudo%2Drelevance Feedback.
We present an algorithm for video retrieval that fuses the decisions of multiple retrieval agents in both text and image modalities. While the normalization and combination of evidence is novel, this paper emphasizes the successful use of negative pseudo-relevance feedback to improve image retrieval performance. Although we have not solved all problems in video information retrieval, the results are encouraging, indicating that pseudo-relevance feedback shows great promise for multimedia retrieval with very varied and errorful data. 1 Video Retrieval from Mixed Text and Image Queries In this paper, we present an algorithm for the task of video retrieval. A query, consisting of a text description plus images or video is posed against a video collection, and relevant shots are to be retrieved. Our system accomplishes this by fusing the retrieval results of multiple retrieval agents. The overall system can be decomposed into several agents, including a text-oriented retrieval agent, which is responsible for finding the text in the speech transcripts [4] and Video OCR [3], a
Information retrieval for OCR documents a content%2Dbased probabilistic correction model.

Modified Logistic Regression An Approximation to SVM and Its Applications in Large%2DScale Text Categorization.
Logistic Regression (LR) has been widely used in statistics for many years, and has received extensive study in machine learning community recently due to its close relations to Support Vector Machines (SVM) and AdaBoost. In this paper, we use a modified version of LR to approximate the optimization of SVM by a sequence of unconstrained optimization problems. We prove that our approximation will converge to SVM, and propose an iterative algorithm called “MLR-CG ” which uses Conjugate Gradient as its inner loop. Multiclass version “MMLR-CG ” is also obtained after simple modifications. We compare the MLR-CG with SVMlight over different text categorization collections, and show that our algorithm is much more efficient than SVMlight when the number of training examples is very large. Results of the multiclass version MMLR-CG is also reported. 1.
Automatically Labeling Video Data Using Multi%2Dclass Active Learning.
Labeling video data is an essential prerequisite for many vision applications that depend on training data, such as visual information retrieval, object recognition, and human activity modeling. However, manually creating labels is not only time-consuming but also subject to human errors, and eventually, becomes impossible for a very large amount of data (e.g. 24/7 surveillance video). To minimize the human effort in labeling, we propose a unified multi-class active learning approach for automatically labeling video data. The contributions of this paper include extending active learning from binary classes to multiple classes and evaluating several practical sample selection strategies. The experimental results show that the proposed approach works effectively even with a significantly reduced amount of labeled data. The best sample selection strategy can achieve more than a 50% error reduction over random sample selection. 
A Faster Iterative Scaling Algorithm for Conditional Exponential Model.
Conditional exponential model has been one of the widely used conditional models in machine learning field and improved iterative scaling (IIS) has been one of the major algorithms for finding the optimal parameters for the conditional exponential model. In this paper, we proposed a faster iterative algorithm named FIS that is able to find the optimal parameters faster than the IIS algorithm. The theoretical analysis shows that the proposed algorithm yields a tighter bound than the traditional IIS algorithm. Empirical studies on the text classification over three different datasets showed that the new iterative scaling algorithm converges substantially faster than both the IIS algorithm and the conjugate gradient algorithm (CG). Furthermore, we examine the quality of the optimal parameters found by each learning algorithm in the case of incomplete training. Experiments have shown that, when only a limited amount of computation is allowed (e.g. no convergence is achieved), the new algorithm FIS is able to obtain lower testing errors than both the IIS method and the CG method. 1.
Modified Logistic Regression An Approximation to SVM and Its Applications in Large%2DScale Text Categorization.
Logistic Regression (LR) has been widely used in statistics for many years, and has received extensive study in machine learning community recently due to its close relations to Support Vector Machines (SVM) and AdaBoost. In this paper, we use a modified version of LR to approximate the optimization of SVM by a sequence of unconstrained optimization problems. We prove that our approximation will converge to SVM, and propose an iterative algorithm called “MLR-CG ” which uses Conjugate Gradient as its inner loop. Multiclass version “MMLR-CG ” is also obtained after simple modifications. We compare the MLR-CG with SVMlight over different text categorization collections, and show that our algorithm is much more efficient than SVMlight when the number of training examples is very large. Results of the multiclass version MMLR-CG is also reported. 1.
Constant Density Displays Using Diversity Sampling.
Consider the problem of removing a subset of images or other visually complex objects from a crowded visualization in order to increase intelligibility. Occlusion severely reduces image intelligibility, and empty space conveys no image information, so optimal choices will tend toward constant density displays. However, previous constant density algorithms are based on global density, which leads to occlusion and empty space if the density is not uniform. This paper introduces an algorithm that considers the layout of individual objects and avoids occlusion altogether. Efficiency concerns are important for dynamic summaries of the Informedia Digital Video Library, which has hundreds of thousands of shots from news stories. Posting multiple queries that take into account parameters of the visualization as well as the original query reduces the amount of work required.
Video retrieval using speech and image information.
Video contains multiple types of audio and visual information, which are difficult to extract, combine or trade-off in general video information retrieval. This paper provides an evaluation on the effects of different types of information used for video retrieval from a video collection. A number of different sources of information are present in most typical broadcast video collections and can be exploited for information retrieval. We will discuss the contributions of automatically recognized speech transcripts, image similarity matching, face detection and video OCR in the contexts of experiments performed as part of 2001 TREC Video Retrieval Track evaluation performed by the National Institute of Standards and Technology. For the queries used in this evaluation, image matching and video OCR proved to be the deciding aspects of video information retrieval.
Web Image Retrieval Re%2DRanking with Relevance Model.
Web image retrieval is a challenging task that requires efforts from image processing, link structure analysis, and web text retrieval. Since content-based image retrieval is still considered very difficult, most current large-scale web image search engines exploit text and link structure to &#034;understand &#034; the content of the web images. However, local text information, such as caption, filenames and adjacent text, is not always reliable and informative. Therefore, global information should be taken into account when a web image retrieval system makes relevance judgment. In this paper, we propose a re-ranking method to improve web image retrieval by reordering the images retrieved from an image search engine. The re-ranking process is based on a relevance model, which is a probabilistic model that evaluates the relevance of the HTML document linking to the image, and assigns a probability of relevance. The experiment results showed that the re-ranked image retrieval achieved better performance than original web image retrieval, suggesting the effectiveness of the re-ranking method. The relevance model is learned from the Internet without preparing any training data and independent of the underlying algorithm  of the image search engines. The re-ranking process should be applicable to any image search engines with little effort.
News video classification using SVM%2Dbased multimodal classifiers and combination strategies.
Video classification is the first step toward multimedia content understanding. When video is classified into conceptual categories, it is usually desirable to combine evidence from multiple modalities. However, combination strategies in previous studies were usually ad hoc. We investigate a meta-classification combination strategy using Support Vector Machine, and compare it with probability-based strategies. Text features from closedcaptions and visual features from images are combined to classify broadcast news video. The experimental results show that combining multimodal classifiers can significantly improve recall and precision, and our meta-classification strategy gives better precision than the approach of taking the product of the posterior probabilities.
Collages as dynamic summaries for news video.
This paper introduces the video collage, a novel effective  interface for browsing and interpreting video collections. The  paper discusses how collages are automatically produced,  illustrates their use, and evaluates their effectiveness as  summaries across news stories. Collages are presentations of text  and images derived from multiple video sources, which provide  an interactive visualization for a set of video documents,  summarizing their contents and providing a navigation aid for  further exploration. The dynamic creation of collages is based on  user context, e.g., an originating query, coupled with automatic  processing to refine the candidate imagery. Named entity  identification and common phrase extraction provides descriptive  text. The dynamic manipulation of collages allows user-directed  browsing and reveals additional detail. The utility of collages as  summaries is examined with respect to other published news  summaries.
A New Probabilistic Model for Title Generation.
Title generation is a complex task involving  both natural language understanding and  natural language synthesis. In this paper, we  propose a new probabilistic model for title  generation. Different from the previous  statistical models for title generation, which  treat title generation as a generation process  that converts the `document representation&#039;  of information directly into a `title  representation&#039; of the same information, this  model introduces a hidden state called  `information source&#039; and divides title  generation into two steps, namely the step of  distilling the `information source&#039; from the  observation of a document and the step of  generating a title from the estimated  `information source&#039;. In our experiment, the  new probabilistic model outperforms the  previous model for title generation in terms  of both automatic evaluations and human  judgments.
The TREC2001 Video Track Information Retrieval on Digital Video Information.
This  paper is not intended to provide a comprehensive picture of the different  approaches taken by the TREC2001 video track participants but instead we give  an overview of the TREC video search task and a thumbnail sketch of the  approaches taken by different groups. The reason for writing this paper is to  highlight the message from the TREC video track that there are now a variety  of approaches available for searching and browsing through digital video  archives, that these approaches do work, are scalable to larger archives and can  yield useful retrieval performance for users. This has important implications in  making digital libraries of video information attainable
Using a probabilistic source model for comparing images.
In this paper, we propose a probabilistic model for image retrieval. To obtain the similarity between the query image I Q and any image I ’ in the collection, this model computes the probability of generating the image I ’ given the observation of the query image I Q. We compared our probabilistic model for image retrieval with a color histogram based image retrieval method and the IBM QBIC image search engine. The evaluation used the 11-hour video retrieval collection (80000 extracted images) and associated queries from the 2001 TREC-10 information retrieval evaluations. The experimental results show that the probabilistic model dramatically outperforms the color histogram based image retrieval method and the IBM QBIC image search engine by 40%. 1.
A Probabilistic Model for Camera Zoom Detection.
Camera motion detection is essential for automated video analysis. We propose a new probabilistic model for detecting zoom-in/zoom-out operations. The model uses EM to estimate the probability of a zoom versus a nonzoom operation from standard MPEG motion vectors. Traditional methods usually set an empirical threshold after deriving parameters proportional to zoom, pan, rotate and tilt. In contrast, our probabilistic model has a solid probabilistic foundation and a clear, simple probability threshold. Experiments show that this probabilistic model significantly out-performs a baseline parametric method for zoom detection in both precision and recall.
Video%2Dcuebik adapting image search to video shots.
We propose a new analysis for searching images in video libraries that goes beyond simple image search, which compares one still image frame to another. The key idea is to expand the definition of an image to account for the variability in the sequence of video frames that comprise a shot. A first implementation of this method for a QBIC-like image search engine shows a clear improvement over still image search. A combination of the traditional still image search and the new video image search provided the overall best results on the TREC video retrieval evaluation data.
Multi%2Dmodal information retrieval from broadcast video using OCR and speech recognition.
We examine multi-modal information retrieval from broadcast video where text can be read on the screen through OCR and speech recognition can be performed on the audio track. OCR and speech recognition are compared on the 2001 TREC Video Retrieval evaluation corpus. Results show that OCR is more important that speech recognition for video retrieval. OCR retrieval can further improve through dictionary-based post-processing. We demonstrate how to utilize imperfect multi-modal metadata results to benefit multi-modal information retrieval.
A wearable digital library of personal conversations.
We have developed a wearable, personalized digital library system, which unobtrusively records the wearer’s part of a conversation, recognizes the face of the current dialog partner and remembers his/her voice. The next time the system sees the same person’s face and hears the same voice, it can replay the audio from the last conversation in compressed form summarizing the names and major issues mentioned. Experiments with a prototype system show that a combination of face recognition and speaker identification can be effective for retrieving conversations.
Video retrieval with multiple image search strategies.

Meta%2DClassification of Multimedia Classifiers.
Combining multiple classifiers is of particular interest in the multimedia systems, since there is usually data of very different types/modalities that should be mined or analyzed. Our wearable &#039;experience collection&#039; system unobtrusively records the wearer&#039;s conversation, recognizes the face of the dialog partner and remembers his/her voice. When the system sees the same person&#039;s face or hears the same voice it can then use a summary of the last conversation with this person to remind the wearer. To correctly identify a person from a mixture of video and audio stream, classification judgments from individual modality classifiers must be combined effectively to yield a more accurate decision. To address the problems of combination strategy in previous studies, a meta-classification strategy using Support Vector Machine is proposed. Preliminary results show that combining different face recognition and speaker identification technology by meta-classification is dramatically more effective than weighted interpolation. Meta-classification is general enough to be applied to any application that needs to combine multiple classifiers without much modification.
Meta%2Dclassification Combining Multimodal Classifiers.

Language model for IR using collection information.
early age of IR where documents are represented by the controlled vocabulary. In this paper, we explore the usage of meta-data information under the framework of language model. We present a new language model that is able to take advantage of the category information for documents to improve the retrieval accuracy. We compare the new language model with the traditional language model over the TREC4 dataset where the collection information for documents is obtained using the kmeans clustering method. The new language model outperforms the traditional language model, which verifies our statement.
Title language model for information retrieval.
In this paper, we propose a new language model, namely, a title language model, for information retrieval. Different from the traditional language model used for retrieval, we define the conditional probability P(Q|D) as the probability of using query Q as the title for document D. We adopted the statistical translation model learned from the title and document pairs in the collection to compute the probability P(Q|D). To avoid the sparse data problem, we propose two new smoothing methods. In the experiments with four different TREC document collections, the title language model for information retrieval with the new smoothing method outperforms both the traditional language model and the vector space model for IR significantly.
Video Classification and Retrieval with the Informedia Digital Video Library System.
This paper is organized in three parts. The first part details some of the lower level shot classification work, the second part describes the ‘manual ’ retrieval systems while the last section details the interactive retrieval system for the Carnegie Mellon University TREC Video Retrieval Track runs. The description of the data can be found elsewhere in the proceedings of the 2002 TREC conference video track overview. Classification In the TREC02 video track, one of the main tasks is to detect various semantic features concepts such as “Indoor/Outdoor”, “People ” etc. This part contains the description of the classification tasks. We submitted runs for the following classification concepts in the TREC 2002 Video Track. To obtain training data, we manually annotated each I-frame of the 23.26 hours feature development collection for each category. a. Outdoors: Segment contains a recognizably outdoor location, i.e., one outside of buildings. Should exclude all scenes that are indoors or are close-ups of objects (even if the objects are outdoor). b. Indoors: Segment contains a recognizably indoor location, i.e., inside a building. Should exclude all scenes that are outdoors or are close-ups of objects (even if the objects are indoor c. Cityscape: Segment contains a recognizably city/urban/suburban setting. d. Monologue: an event in which a single person is at least partially visible and speaks for a long time without
Title Generation Using a Training Corpus.

Learning to Select Good Title Words An New Approach based on Reverse Information Retrieval.
In this paper, we show how we can learn to  select good words for a document title. We  view the problem of selecting good title words  for a document as a variant of an Information  Retrieval problem. Each title word is treated as  a &#034;document&#034; and selection of appropriate title  words as finding relevant &#034;documents&#034;. Based  on our training collection consisting of 40,000  document and title pairs, we learn the  &#034;document&#034; representations for all the title  words and apply these learned representations  to select appropriate title words over 10,000  test documents. Compared to other learning  approaches, namely K nearest neighbor  approach, a Nave Bayesian approach and a  variant of a machine translation model, we  find that our approach is significantly better as  indicated by the F1 metric.
Title Generation for Machine%2DTranslated Documents.
In this paper, we present and compare automatically generated titles for machine-translated documents using several different statistics-based methods. A Naïve Bayesian, a K-Nearest Neighbour, a TF-IDF and an iterative Expectation-Maximization method for title generation were applied to 1000 original English news documents and again to the same documents translated from English into Portuguese, French or German and back to English using SYSTRAN. The AutoSummarization function of Microsoft Word was used as a base line. Results on several metrics show that the statisticsbased methods of title generation for machinetranslated documents are fairly language independent and title generation is possible at a level approaching the accuracy of titles generated for the original English documents. 1.
Demonstration of hierarchical document clustering of digital library retrieval results.
As digital libraries grow in size, querying their contents will  become as frustrating as querying the web is now. One remedy  is to hierarchically cluster the results that are returned by  searching a digital library. We demonstrate the clustering of  search results from Carnegie Mellon&#039;s Informedia database, a  large video library that supports indexing and retrieval with  automatically generated descriptors.
Meta%2Dscoring Automatically Evaluating Term Weighting Schemes in IR without Precision%2DRecall.
In this paper, we present a method that can automatically evaluate performance of different term weighting schemes in information retrieval without resorting to precision-recall based on human relevance judgments. Specifically, the problem is: given two document-term matrixes generated from two different term weighting schemes, can we tell which term weighting scheme will performance better than the other? We propose a meta-scoring function, which takes as input the document-term matrix generated by some term weighting scheme and computes a “goodness ” score from the documentterm matrix. In our experiments, we found out that this score is highly correlated with the precision-recall measurement for all the collections and term weighting schema we tried. Thus, we conclude that our meta-scoring function can be a substitute for the precision-recall measurement that needs relevance judgments of human subject. Furthermore, this meta-scoring function is not limited only to text information retrieval can be applied to fields such as image and DNA retrieval.
Video Retrieval with the Informedia Digital Video Library System.
information extraction from video and audio content. Over a terabyte of online data was collected, with automatically generated metadata and indices for retrieving videos from this library. The architecture for the project was based on the premise that real-time constraints on library and associated metadata creation could be relaxed in order to realize increased automation and deeper parsing and indexing for identifying the library contents and breaking it into segments. Library creation was an offline activity, with library exploration by users occurring online and making use of the generated metadata and segmentation. The goal of the Informedia interface was to enable quick access to relevant information in a digital video library, leveraging from derived metadata and the partitioning of the video into small segments. Figure 1 shows the IDVLS interface following a query. In this figure, a set of results is displayed at the bottom. The display includes a window containing a headline, and a pictorial menu of video segments each represented with a thumbnail image at approximately  resolution of the video in the horizontal and vertical dimensions. The headline window automatically pops up whenever the mouse is positioned over a result item; the headline window for the first result is shown. IDVLS also supports other ways of navigating and browsing the digital video library. These interface features were essential to deal with the ambiguity of the derived data generated by speech recognition,
Automatic title generation for EM.

Data Analysis for a Multimedia Library.

Complementary Video and Audio Analysis for Broadcast News Archives.

Adjustable Filmstrips and Skims as Abstractions for a Digital Video Library.
Filmstrips and video skims are two presentation schemes for abstracting information in a digital video segment. Filmstrips present information all at once in a static form, while video skims are played and disclose information temporally. This paper discusses the evolution of the filmstrip and skim interfaces in the Informedia Digital Video Library. Filmstrips are commonly deployed as interfaces for video and image libraries, but we found initial Informedia filmstrips and skims received little use. We discuss the interface considerations motivating the redesign of filmstrips and skims to adjust their presentations dynamically based on user context and preference.

Improving Acoustic Models with Captioned Multimedia Speech.

CMU Spoken Document Retrieval in Trec%2D8 Analysis of the role of Term Frequency TF.
this paper we attempted tf*idf a more theoretic interpretation and point out a possible reason why multiplying tf directly with idf causes the poor performance with the given interpretation. We came up with a word histogram based method of integrating tf and idf into one factor which is log(1/p&#039;) and p&#039; is the probability that a document has at least tf occurrences of a particular word. We have several success over tf*idf and (ln(tf)+1)*idf and fail to compete with (ln(ln(tf)+1)+1)*idf. We think the failure is due to the &#034;ending effect&#034; and we will pursue the problem further in the future.
Informedia Experience%2Don%2DDemand Capturing
The Informedia Experience-on-Demand system uses speech, image, and natural language processing combined with GPS information to capture, integrate, and communicate personal multimedia experiences. This paper discusses an initial prototype of the EOD system.
Lessons Learned from Building a Terabyte Digital Video Library.
y boundaries manually for the documentary  video; in our subsequent experiments we looked at generating  story boundaries automatically. The stories, or  video segments, averaged a few minutes in length, so  that the entire library contained more than 40,000  video segments.  We used artificial intelligence techniques to create  metadata, the data that describes video content. We  found that all the AI techniques we used were applicable  to both the news corpus and the documentary  corpus. These techniques included speech recognition,  image processing, and information retrieval. We  learned that by integrating across these three areas we  were able to compensate for limitations in accuracy,  coverage, and communicative power.  Speech processing  We analyzed the audio component of the video with  the CMU Sphinx speech recognition system.  1  This created  a complete transcript for text-based retrieval  from the speech and aligned existing imperfect trans

Learning to Recognize Speech by Watching Television.

Topic Labeling of Broadcast News Stories in the Informedia Digital Video Library.

Story Segmentation and Detection of Commercials in Broadcast News Video.
The Informedia Digital Library Project [Wactlar96] allows full content indexing and retrieval of text, audio and video material. Segmentation is an integral process in the Informedia digital video library. The success of the Informedia project hinges on two critical assumptions: that we can extract sufficiently accurate speech recognition transcripts from the broadcast audio and that we can segment the broadcast into video paragraphs, or stories, that are useful for information retrieval. In previous papers [Hauptmann97, Witbrock97, Witbrock98], we have shown that speech recognition is sufficient for information retrieval of pre-segmented video news stories. In this paper we address the issue of segmentation and demonstrate that a fully automatic system can extract story boundaries using available audio, video and closed-captioning cues. The story segmentation step for the Informedia Digital Video Library splits full-length news broadcasts into individual news stories. During this phas...
Experiments in Spoken Document Retrieval at CMU.
We describe our submission to the TREC-7 Spoken Document Retrieval (SDR) track and the speech recognition and information retrieval engines. We present SDR evaluation results and a brief analysis. A few developments are also described in greater detail including:  . A new, probabilistic retrieval engine based on language models.  . A new, TFIDF-based weighting function that incorporates word error probability.  . The use of a simple confidence estimate for word probability based on speech recognition lattices. Although improvements over a development test set were promising, the new techniques failed to yield significant gains in the evaluation test set. 1. The SDR Data and Task  The entire set of speech data for the 1998 TREC-7 spoken document retrieval track consisted of 153 hours of broadcast news, approximately 80 for training and 73 for testing. The data had been segmented into stories and manually transcribed. In the test set, there were three &#034;versions&#034; of the data available: A manually generated transcript, speech recognition transcripts based on IBM and CMU recognizers, and the raw audio data, to be transcribed by our own recognizer. The entire training set was used to train acoustic models for the speech recognition system. The remainder was held out as unseen test data. There were about 3245 stories in the training data set and 2866 in the test set. To develop and debug the system, the TREC-6 evaluation set was used in a Known-Item Retrieval system -- where every query has only one document assigned as relevant. In our experiments on the evaluation test set, the average precision of the retrieval for each of the relevant documents was used to judge the quality of the retrieval. However, since relevance judgements were not available for the development test se...
Speech Recognition for a Digital Video Library.
The standard method for making the full content of audio and video material searchable and is to annotate it with human-generated meta-data that describes the content in a way that the search can understand, as is done in the creation of multimedia CD-ROMs. However, for the huge amounts of data that could usefully be included in digital video and audio libraries, the cost of producing this meta-data is prohibitive. In the Informedia Digital Video Library, the production of the meta-data supporting the library interface is automated using techniques derived from artificial intelligence (AI) research. By applying speech recognition together with natural language processing, information retrieval and image analysis, an interface has been produced that helps users locate the information they want and navigate or browse the digital video library more effectively. Specific interface components include automatic titles, filmstrips, video skims, word location marking and representative frames f...
Using Words and Phonetic Strings for Efficient Information Retrieval from Imperfectly Transcribed Spoken Documents.

Artificial Intelligence Techniques in the Interface to a Digital Video Library.

Experiments in Spoken Document Retrieval at CMU.
We describe our submission to the TREC-7 Spoken Document Retrieval (SDR) track and the speech recognition and information retrieval engines. We present SDR evaluation results and a brief analysis. A few developments are also described in greater detail including:  . A new, probabilistic retrieval engine based on language models.  . A new, TFIDF-based weighting function that incorporates word error probability.  . The use of a simple confidence estimate for word probability based on speech recognition lattices. Although improvements over a development test set were promising, the new techniques failed to yield significant gains in the evaluation test set. 1. The SDR Data and Task  The entire set of speech data for the 1998 TREC-7 spoken document retrieval track consisted of 153 hours of broadcast news, approximately 80 for training and 73 for testing. The data had been segmented into stories and manually transcribed. In the test set, there were three &#034;versions&#034; of the data available: A manually generated transcript, speech recognition transcripts based on IBM and CMU recognizers, and the raw audio data, to be transcribed by our own recognizer. The entire training set was used to train acoustic models for the speech recognition system. The remainder was held out as unseen test data. There were about 3245 stories in the training data set and 2866 in the test set. To develop and debug the system, the TREC-6 evaluation set was used in a Known-Item Retrieval system -- where every query has only one document assigned as relevant. In our experiments on the evaluation test set, the average precision of the retrieval for each of the relevant documents was used to judge the quality of the retrieval. However, since relevance judgements were not available for the development test se...
Demonstration of a Reading Coach that Listens.
use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Demonstration of a Reading Coach that Listens
Speech for Multimedia Information Retrieval.
We describe the Informedia  tm  News-on-Demand system. News-on-Demand is an innovative example of indexing and searching broadcast video and audio material by text content. The fully-automatic system monitors TV news and allows selective retrieval of news items based on spoken queries. The user then plays the appropriate video &#034;paragraph&#034;. The system runs on a Pentium PC using MPEG-I video compression and the Sphinx-II continuous speech recognition system [6].
A Reading Coach that Listens %28Edited%29 Video Transcript.

A Prototype Reading Coach that Listens.
use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Demonstration of a Reading Coach that Listens
A Prototype Reading Coach that Listens Summary of Project LISTEN.

Survey of Current Speech Technology.
This article describes two technologies, speech recognition and speech synthesis, that manipulate speech in terms of its information content. Recognition is the transformation of human speech into text to be used literally (e.g., for dictation) or interpreted as commands to control applications. Synthesis allows the generation of spoken utterances from text. Synthesis is desirable when a large number of utterances must be available or when message content is unpredictable, requirements that make pre-recording of speech impractical. The technologies covered in this article are of particular interest because they support direct communication between humans and computers through a mode that humans commonly use for communication amongst themselves and at which they are highly skilled. Other speech technologies of note, not discussed here, include speaker recognition (automatically establishing a speaker&#039;s identity) as well as speech editing and indexing (the manipulation of speech without the extraction of linguistic information). 1 RECOGNITION
Towards a Reading Coach that Listens Automated Detection of Oral Reading Errors.
use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Gestures with Speech for Graphic Manipulation.
An experiment was conducted with people using gestures and speech to manipulate graphic images on a computer screen. A human was substituted for the recognition devices. The analysis showed that people strongly prefer to use both gestures and speech for the graphics manipulation and that they intuitively use multiple hands and multiple fingers in all three dimensions. There was surprising uniformity and simplicity in the gestures and speech. The analysis of these results provides strong encouragement for future development of integrated multi-modal interaction systems.  KEYWORDS: Gestures, speech, manipulation, input,  graphics.  INTRODUCTION  This paper presents a study of people intuitively using gestures and speech to communicate with computers. Even though we have seen an explosion in the capabilities of computers to present output, the bandwidth of the input channel to a computer is still very limited. Aside from the keyboard, the input devices that are available to users, such as...
From Syntax to Meaning in Natural Language Processing.

Layering Predictions Flexible Use of Dialog Expectation in Speech Recognition.
for relaxing contextual constraints when they are violated. When computer speech recognition is used for problem  In order to do this, we organize constraints into sets that are  solving or any plan based task, predictable features of the  successively more restrictive, called &#034;layers&#034;. When some  user&#039;s behavior may be inferred and used to aid the  constraints are violated, we use the non-violated constraints  recognition of the speech input. The MINDS system  from a less restrictive layer to reduce search space.
High Level Knowledge Sources in Usable Speech Recognition Systems.

Using Dialog%2DLevel Knowledge Sources to Improve Speech Recognition.
We motivate and describe an implementation of the MINDS speech recognition system. MINDS uses knowledge of dialog structures, user goals and focus in a problem solving situation. The knowledge is combined to form predictions which translate into dynamically generated semantic network grammars. An experiment evaluated recognition accuracy given different levels of knowledge as constraints. Our results show that speech recognition accuracy improves dramatically, when the maximally constrained dynamic network grammar is used to process the speech input signal.
Talking to Computers An Empirical Investigation.

Parsing Spoken Language A Semantic Caseframe Approach.
Parsing spoken input introduces serious problems not present in parsing typed natural language. In particular, indeterminacies and inaccuracies of acoustic recognition must be handled in an integral manner. Many techniques for parsing typed natural language do not adapt Well to these extra demands. This paper describes an extension of semantic caseframe parsing to restricted-domain spoken input. The semantic caseframe grammar representalien is the same as that used for earlier work on robust parsing of typed input. Due to the uncertainty inherent in speech recognition, the caseframe grammar is applied in a quite different way, emphasizing island growing from caseframe headers. This radical change in application is possible due to the high degrbe of abstraction in the caseframe representation. The eppro,&#039;:tch presented was tested successfully in a preliminary implementation.
